Add IO runtime handle to postgres connector#549
Merged
phillipleblanc merged 1 commit intodatafusion-contrib:mainfrom Feb 26, 2026
Merged
Add IO runtime handle to postgres connector#549phillipleblanc merged 1 commit intodatafusion-contrib:mainfrom
phillipleblanc merged 1 commit intodatafusion-contrib:mainfrom
Conversation
Contributor
Author
|
cc: @phillipleblanc who reviewed the other postgresSQL PR |
phillipleblanc
approved these changes
Feb 26, 2026
Collaborator
phillipleblanc
left a comment
There was a problem hiding this comment.
Makes sense to me, thanks!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional
with_io_runtime(handle)builder method toPostgresConnectionPool, allowing callers to route connection acquisition (and the background socket-IO task thatbb8spawns internally) onto a dedicated tokio IO runtime.This is the IO runtime segregation follow-up noted in #548.
Problem
When a query engine runs DataFusion plans and Postgres IO on the same tokio runtime, CPU-bound work (plan optimization, aggregation, shuffles) starves IO tasks, causing connection timeouts, health-check failures, and unpredictable tail latency. This is the same class of issue fixed in #450 (non-blocking connection check), but at the connection-pool level rather than the DNS-lookup level.
Solution
io_handle: Option<Handle>toPostgresConnectionPoolwith_io_runtime(handle: Handle) -> Selfbuilder methodpool.get_owned()is dispatched viahandle.spawn(...)so the connection and its background task land on the IO runtimerun_async_with_tokio/ direct.await)The returned
Clientcommunicates with its background task via channels, so it works correctly from any runtime after acquisition.Why this pattern
DataFusion's own documentation explicitly recommends runtime segregation:
The pattern of accepting an optional
Handleat construction time is converging across the ecosystem:object_storeSpawnedReqwestConnector::new(handle)Changes
core/src/sql/db_connection_pool/postgrespool.rsIoRuntimeError(wrapstokio::task::JoinError)io_handle: Option<Handle>onPostgresConnectionPoolwith_io_runtime(handle)connect()andconnect_direct()dispatch through the handle when presentcore/tests/postgres/mod.rswith_io_runtime(), and runs a query end-to-endUsage
Without
with_io_runtime(), behavior is unchanged.Test plan
cargo build --features postgrescompilestest_postgres_io_runtime_segregationintegration test verifies the IO-handle path end-to-endio_handleisNone)